1. Field of the Invention
The present invention relates to computer-based technology in assessment particularly for the licensing and certification of professionals such as architects, civil engineers, aeronautical engineers, mechanical engineers, naval engineers, interior design, landscape design, architectural design, etc. For assessment of architects, the system includes tools and methods for item creation, computer programs for computerized item presentation, and programs for automatically scoring test responses by computer.
The purpose of assessment in licensing and certification is to make accurate and reliable decisions as to whether a candidate has met certain standards of competent performance, ordinarily involving a range of higher order cognitive skills as well as the mastery of an extensive knowledge base.
The National Council of Architectural Registration Boards (NCARB) each year prepares the Architecture Registration Exam (ARE) to register entry-level candidates for the practice of architecture. This registration examination is used by 55 member jurisdictions and most Canadian provinces as the basis for initial licensure and reciprocity. The goal of the Architecture Registration Exam is to ensure that only architectural candidates proven to meet a competency level established to protect the health safety and welfare of the public are allowed to practice in the profession.
In the past, the examination consisted of several multiple-choice tests and graphics tests administered annually over a four-day period, with the graphics tests administered one additional time each year. The core functions of architectural practice, namely, site design and building design, are accorded fundamental importance in the graphics tests. Since the purpose of the examination is to protect the public's health safety and welfare, the component tests focus on essential elements of competent practice, rather than on areas like aesthetics of design. Nonetheless, candidates have some opportunity to display creativity by accomplishing tasks requiring problem-solving under constraints.
2. Description of Related Art
While the ARE was originally entirely a paper-and-pencil based test, recently multiple-choice parts of the examination have been administered by computer in a format patented by Educational Testing Service, inventors' assignee, and called Computerized Mastery Testing. That patent, U.S. Pat. No. 5,059,127, entitled COMPUTERIZED MASTERY TESTING SYSTEM, A COMPUTER ADMINISTERED VARIABLE LENGTH SEQUENTIAL TESTING SYSTEM FOR MAKING PASS/FAIL DECISIONS, is hereby incorporated herein.
In the paper-and-pencil implementation of the building design graphics test, a candidate for registration spends 12 hours developing plans and drawing related sections and elevations. The output of this paper-and-pencil test is a pad of four drawings. Over a hundred architect/jurors meet for three days twice a year to grade these drawings using a holistic grading process.
While in the paper-and-pencil implementation of the site design graphics test, a candidate spends two and three-quarters hours developing solutions to site design vignettes. These drawings are also graded holistically in sessions twice a year involving nearly seventy additional architect/jurors.
Figural Response.
The present invention is different from another testing methodology called "figural response" invented and patented by applicants' assignee Education Testing Service. See, e.g., U.S. Pat. No. 5,011,413, entitled MACHINE-INTERPRETABLE FIGURAL RESPONSE TESTING.
Figural response requires the candidate to respond to a graphic question or problem presented on a video display, by drawing lines or other simple geometric figures, positioning arrows or other markers on the graphic, or repositioning elements of the graphic, etc. While figural response may appear to be similar to the simulation methodology of the present invention, differs in many respects--some of which follow. The tasks required of the candidate in figural response were necessarily simpler resulting in problems of a relatively limited nature with but one correct answer for each figural response item.
Figural response relies on bitmap graphics; the present invention relies on a completely different method of image representation called vector graphics. Two main benefits of a vector based approach to graphics are that all the things shown on the screen to the candidate are represented as discrete and insular "objects" inside the computer, rather than as mere regions of a bitmap. A vector/object approach makes building a rich and complex scoring system a much less arduous task. Similarly, a vector/object approach makes the creation of new objects much easier to do.
Though both tests are graphical in presentation the nature of the tasks required by the test taker are very different. Figural response required the test taker to use a mouse to select one of several pictures, or to draw a line connecting two related images. Each figural response item has a single correct answer. The present invention asks the candidate to construct a complete architectural drawing on a nearly blank background. Because of the level of detail allowed, and a conscious effort on the part of the developers to allow for individual style and creativity, there can be an infinite number of equally valid solutions to most of the ARE items.
The figural response items were limited to fixed screen sizes whereas the items of the present invention can be used at any resolution from 480.times.600 on up. This is because vector images can scale themselves to fit various screen or area requirements much more easily than bitmaps can.
Finally, although figural response items are scored by computer the scoring programs are much simpler because the tasks required of the candidate were much simpler. The simplicity of the task also means that there is only one correct answer to a figural response question, as noted above. Despite that simplicity, there was very little reuse of code across figural response scoring programs; every item needed its own scoring program. The present invention uses the same code for an entire vignette item family. Currently a vignette family may consist of twenty-four individual items which are equivalent in difficulty and the skills tested.
Difficulties with prior art methods.
There are several inherent difficulties with the ARE's paper-and-pencil method of test administration. In particular, the paper-and-pencil ARE could only be scheduled infrequently--twice a year. Many weeks were needed for scores to be reported to candidates because of the time required to plan and host a "convention-style" grading where many human graders were flown into a central grading location.
The paper and pencil ARE exams were scored in large convention style settings. Such gradings involve finding skilled architects who can take time off from their work to serve as graders, transporting them to a central location, making hotel arrangements for all the people involved, and hiring and training temporary clerical staff to manage a large volume of secure paper.
Humans grade less consistently than computers and are harder to monitor. Human grades rely on a gross holistic view of a solution which incorporates much less detail and precision than computer scores.
Computer generated ratings may be superior to human scores because testing experts and experts skilled in the art of architecture can manipulate the relative impact that each feature of a solution that is analyzed has on the solution's final score.